Transmission of data between a client and a server typically begins with the client and server performing a sequence of handshakes to establish a connection. In a Transmission Control Protocol (TCP) handshake, SYN, SYN-ACK, and ACK messages are transmitted between the client and server to establish the connection. The SYN message initiates the communication between the server and client while the SYN-ACK and ACK messages establish the connection. Traditionally, the server generates a TCP control block when a SYN message is received. The SYN message includes synchronization data to synchronize the client with the server. For example, the synchronization data specified by the client. A TCP control block maintains data about a potential connection between a client and server such as the socket numbers, points to buffers, bytes received and acknowledged, etc. The TCP control block is maintained through the duration of the connection and is deleted when the connection ends.
Due to memory limitations of the server, a malicious client can attack a server by using a SYN flood attack. A SYN flood attack is when a malicious client transmits multiple dummy SYN messages to the server. Since each SYN message received generates a TCP control block, the server can run out of memory as it maintains TCP control blocks for these dummy connections. As a result, the server will be unable to service connection requests from legitimate clients.
A SYN cache technique may be a technique to protect against SYN flood attacks. In the SYN cache technique, instead of initially creating a TCP control block for each connection request, the server hashes some or all of the synchronization data to generate a hash value. The hash value is stored in a hash table on the server used to track potential connections with clients. The client and server then engage in further handshake communications to verify the connection using the hash value stored in the hash table to verify the handshake. Once verified, the connection is established and the TCP control block is generated for the established connection. Also, the hash value is deleted from the hash table. Therefore, the TCP control block is generated only for established connections, thus minimizing system resources that would have been wasted from a SYN flood attack.
Because the server generates the hash value using client specific data, the client can possibly control where the server stores the synchronization data in the hash table. This can hinder the performance of the hash table, particularly if the hash keys are places in the same row (i.e., hash bucket) of the hash table. For example, each hash bucket may contain a limited number of entries for connection requests. Once the hash bucket is filled with hash values, the server cannot service any additional SYN messages directed to that hash bucket without deleting existing entries in the hash table. This may cause performance problems as the server may drop legitimate connection requests. Thus, a malicious client may attempt to fill the same hash bucket of the hash table by specifying client specific data used by the server to determine where to store the hash value in the hash table.